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Background 

Student-level, school-based longitudinal evaluations of educational practices and interventions 
often encounter study attrition when students move away from, or drop out of, study schools. 
This mobility-induced attrition decreases sample size, weakening the study’s power to detect a 
treatment effect. While the research methodology literature includes empirical benchmarks for 
effect sizes (Hill, Bloom, Black, & Lipsey, 2008; Bloom, Hill, Black, & Lipsey, 2008) and 
intraclass correlations (Hedges & Hedberg, 2007; Schochet, 2008) to help researchers determine 
adequate sample sizes through power analysis, it does not include similar benchmarks to assist 
proper planning for attrition. If a power analysis does not take possible attrition into account, the 
planned study could be underpowered. Figure 1 shows how student attrition can affect the 
minimum detectable effect size (MDES) under different study designs. 


Purpose 

This paper describes and demonstrates a new online resource developed to help researchers 
proactively plan for student attrition: the Student Attrition Lookup Tool (SALT). With SALT, 
researchers can access student mobility estimates for different student and school subpopulations 
and for different transition periods from kindergarten to grade 12. The paper has three main 
objectives: (1) describe the data and methods behind SALT; (2) demonstrate how to navigate 
SALT; and (3) demonstrate how to use student attrition benchmarks in a power analysis. The 
paper is primarily intended for applied researchers planning school-based studies that follow 
students over time. 


Data 

SALT is based on four longitudinal data sets from the National Center for Education Statistics 
(NCES). The use of multiple data sets allows us to examine rates of student mobility for students 
across a range of grade levels—with two data sets covering mobility during the elementary 
school years and two data sets covering mobility during the high school years—and present 
findings across different U.S. public school student populations relevant to applied researchers. 
Important aspects of each data set are provided in Table 1. 


Methods 

With the four NCES surveys, we constructed analytic files to examine student mobility across 
six transition periods (see Figure 2). To examine student mobility during each transition period 
accurately, we restricted the analytic files to students in the appropriate longitudinal follow-up 
waves and to public schools with a grade range that includes the transition period (e.g., the 
ECLS-K:98 grade 1 to 3 analytic file is restricted to schools with grades | and 3). 
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Since applied researchers increasingly utilize data from district or state student information 
systems, we estimated three types of student mobility rates (for each transition period) to reflect 
three different levels of data coverage (see Figure 3): 

* Any mobility (left baseline school / outside school data) 

= Left the district, state, or public school system (outside district data) 

= Left the state or public school system (outside state data, including dropouts) 


We estimated mobility rates for the total sample, as well as for student and school subgroups (see 
Table 2 and Table 3). When calculating student mobility rates, we used Taylor-series 
linearization to account for survey weights and the multistage nature of the survey designs. 
Using the survey sampling weights allows us to calculate mobility rates that are generalizable to 
the U.S. public school student population. 


Results 

Researchers can access SALT through an interactive website that is currently under 
development, and will be publicly available before the SREE conference. Through the SALT 
website, researchers can get student mobility rates for any of the student or school subgroups 
listed in Tables 2 and 3, as well as different subgroup combinations (e.g., students with low math 
achievement in urban schools). A sample screenshot of the SALT website is provided in Figure 
4. Users have the option to add confidence intervals to the graph, and can request a table with the 
mobility rate estimates and standard errors. 


To demonstrate how researchers can use SALT in power analyses, we walk through an example 
power analysis for an individual random assignment design. We use the following formula to 
determine how student attrition influences the minimum required sample size (MRSS) for a 


given MDES: 
2 _p2 
mrss, = (“aeet) (5*=) (45). 


where My ¢ af is the multiplier for a given alpha-level (type I error), beta-level (type II error), 
and degrees of freedom (df); R? is the proportion of variance explained by covariates; P is the 
proportion of units in the treatment group; and w is the student attrition rate. 


Since the student attrition rate is unknown prior to conducting the study, we can get estimates of 
the rate using SALT. On the SALT website, we use the pull-down menus to request student 
mobility estimates that align with our sample of interest. Results from this query are displayed in 
Figure 4. Depending on the type of data collection, the expected student mobility rate is 26% 
(data from schools), 18% (data from districts), or 12% (data from state). 


By plugging these attrition estimates into the above MRSS equation (and assuming R? = .50 and 
P=.50), we get the power analysis results presented in Figure 5. If we ignored attrition and 
wanted an MDES of 0.10, we would target a sample size of 1,576 students. That would, 
however, result in an underpowered study given attrition. If we only collect data from students 
who stay in the study schools through 11" grade, the actual sample size needed for an MDES of 
0.10 is 2,130 students. If, however, we can get student data from a statewide database, the 
sample size needed for an MDES of 0.10 is 1,922 students. With this information, we can 
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determine whether having 208 fewer students in the study (2,130 — 1,922) is worth the effort of 
gaining access to statewide data, rather than collecting data directly from the study schools. 


The paper will also demonstrate how attrition benchmarks from SALT can be incorporated into 
existing power analysis software, such as PowerUp! (Dong & Maynard, 2013), and discuss the 
use of student attrition benchmarks in block and cluster randomized designs. 


Conclusions 

This paper addresses a limitation in current power analyses by providing researchers with 
information about natural rates of mobility-based attrition. Student mobility represents the 
attrition rate one can expect under conditions in which factors such as consent and survey 
response are not an issue. In addition, mobility is a major source of attrition in studies that rely 
on extant administrative data (e.g., state assessment data). By making student mobility rates across 
different student and school characteristics, as well as different degrees of mobility that correspond to 
different data collection options accessible, researchers can proactively plan for normative attrition 
rates in longitudinal studies. 
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Tables and Figures 


Table 1. Summary of data sources and analytic file sample sizes 


Data Transition period Survey waves _ | Analytic file sample 

source (child weight) 

ECLS-K:10 | Kindergarten (spring Wave 2 to wave | 10,300 students in waves 2 to 4 
2011) to grade 1 4 (W4C4P_20) longitudinal sample and in a wave 2 public 
(spring 2012) school with kindergarten and grade 1 

ECLS-K:10 | Kindergarten (spring Wave 2 to wave | 9,350 students in waves 2 to 6 longitudinal 
2011) to grade 2 6 (W6C6P_20) sample and in a wave 2 public school with 
(spring 2013) kindergarten and grade 2 

ECLS-K:98 | Grade 1 (spring 2000) | Wave 4 to wave | 10,220 students in waves 4 to 5 
to grade 3 (spring 5 (C45CWO) longitudinal sample and in a wave 4 public 
2002) school with grades 1 and 3 

ECLS-K:98 | Grade 1 (spring 2000) | Wave 4 to wave | 7,280 students in waves 4 to 6 longitudinal 
to grade 5 (spring 6 (C456CWO) sample and in a wave 4 public school with 
2004) grades 1 and5 

HSLS Grade 9 (fall 2009) to Wave 1 to wave | 15,160 students in waves 1 to 2 
grade 11 (spring 2012) | 2 (W2W1STU) longitudinal sample and in a wave 1 public 

school: 

ELS Grade 10 (spring 2002) | Wave 1 to wave | 10,900 students in waves 1 to 2 
to grade 12 (spring 2 (FIPNLWT) longitudinal sample and in a wave 1 public 
2004) school 


NOTE: All sample sizes are rounded to the nearest 10 per IES publication policy. 


Table 2. Student subgroups included in SALT, by data source 


Characteristic Subgroups Included in SALT ECLS-K:10 | ECLS-K:98 HSLS ELS 

Gender Female Y 2 V y 
Male 

Race African American 
Asian/Pacific Islander Y yf e Y 
Latino/Hispanic 
White 

SES (NCES-defined Low (bottom two quintiles) Y Z 2 se 

quintiles) High (top two quintiles) 

Baseline reading Low (bottom two quintiles) Y af Y 

achievement High (top two quintiles) 

Baseline mathematics | Low (bottom two quintiles) Y Z V Y 

achievement High (top two quintiles) 

Student disability Student with a disability us Y Y Y 

status Student without a disability 

Home or native English is home/native language 

language English is not home/native v v a 
language 

NOTE: SES = socioeconomic status. 
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Table 3. School subgroups included in SALT 


Characteristic 


Subgroups Included in SALT 


Title | status 


Title | school 
Not a Title | school 


Geographic region 


Northeast 
Midwest 
South 
West 


Location 


City 
Suburb 
Rural or town 


School size (enrollment) 


Small (less than 500 students for elementary schools; 
1,500 for high schools) 

Large (500 or more students for elementary schools; 1,500 
for high schools) 


Concentration of underrepresented 
minorities 


Low (less than 25% of total enrollment) 
Medium (25-49% of total enrollment) 
High (50% or more of total enrollment) 


Concentration of students eligible for 
Free/Reduced Price Lunch program 


Low (less than 25% of total enrollment) 
Medium (25-49% of total enrollment) 
High (50% or more of total enrollment) 


District size (number of schools in the 
district with a grade range that 
includes the transition period) 


Small (less than 5 schools) 

Medium (5-19 schools for elementary schools; 5-9 schools 
for high schools) 

Large (at least 20 schools for elementary schools; 10 for 
high schools) 
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Figure 1. Relationship between student attrition and MDES for select research designs 
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NOTE: Example power analysis parameters set to detect an MDES of 0.15 (significance level alpha = 0.05, 
statistical power beta = 0.80) under a condition with no student attrition. 

IRA = individual random assignment design 

BIRA = block individual random assignment design 

CRA = cluster random assignment design 

Model numbers in parentheses refer to the model numbering used in Dong & Maynard (2013). 
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Figure 2. Transition periods included in SALT 


source |__| ast | 2nd | 3rd | ath | sth | ms | oth | 20th | 23th | 22th | 
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NOTE: All surveys were administered during the spring of the school year, with the exception of the HSLS ninth- 
grade survey, which was administered during the fall of grade 9. 


Figure 3. Mobility classification for SALT 
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Figure 4. Sample screenshot from preliminary version of SALT website 
How to use SALT 


Use the menus on the left to select the student attrition rates you are interested in for your study. To begin the process, identify the transition period of interest. Then choose the type of subgroup interaction and the specific school and/or student characteristics for your 
subgroup of interest. The estimated student attrition rates for that subgroup are displayed in a bar graph and a table. 


Select Transition Period graph table 
9to 11 [z| 50- 
Select Interaction Type 
school-by-student interaction | ad 
Z 
School Characteristic 
Soo. 
City location i-| a 
3 
Student Characteristic Eo. 179% 
Low math achievement (bottom two quintiles) -| 5 
Include Confidence interval 10- 


Any mobility District mobilty’ 


Notes: Mobility rates based on fewer than 10 students are suppressed. See the technical report for details about subgroup definitions and how the student mobility rates were estimated. (coming soon) 
Any mobility = Any move out of the student's original school (attrition when only collecting data from specific study schools) 

District mobility = Move to a school in a different school district (attrition when only collecting data from specific school districts)* 

State mobility = Move outside the state public school system, including moves to a private school or dropping out of school (attrition when only collecting data from specific states) 


* Data for the grade 10 to 12 transition period did not include enough information to distinguish between students who transferred to another school within the same district and students who transferred to another district within the same state. As a result, we are not 
able to provide mobility rates for grade 10 to 12 district mobility. 
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